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ABSTRACT 


Across the social sectors, a growing number of 
organizations are embracing continuous improvement 
approaches and organizing themselves as networks in 
order to attack complex problems. To succeed, networks 
must learn quickly by studying their own practices, 
continuously adapting to changing circumstances within 
their organizations and in the broader environment, and 
incorporating this learning into their ongoing work. In this 
paper, we describe an integrated approach—which we 
call Evidence for Improvement (EFI)—that explains how 
a variety of tools and practices drawn from diverse forms 
of program evaluation can inform the leaders of these 
networks in advancing productive change. We propose 
that improvement networks can be conceptualized and 
measured using a three-level nested model composed 

of a working theory of improvement, an improvement 
enterprise, and environmental contexts. 


As they engage in inquiry targeted at each level, those with 
evaluative expertise should become authentic partners 
with improvement leaders and fully align their analytic 
efforts with the improvement activities and social contexts 
of the network. The EFI approach we describe is designed 
to enhance a network’s internal learning processes and, 

in turn, lead to more positive impacts for educators and 
students. It has implications for improvement practitioners, 
evaluators, and funders. 
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Across the social sectors, a growing number of own practices, continuously adapting to changing 
organizations are embracing continuous improvement circumstances within their organizations and in the 
approaches to attack complex problems. Calls for broader environment, and incorporating this learning 
continuous improvement in education appear into their ongoing work.’ 

throughout the Every Student Succeeds Act (ESSA), 
and prominent philanthropies have pledged to 
support improvement networks that use local data to THIS PAPER IN CONTEXT 


drive positive change in important outcomes.’ These This document offers an integrated 


developments suggest that an increasing number 


erspective on how a variety of tools 
of social initiatives will be using improvement et y 


methodologies and organizing themselves as networks, and practices, drawn from diverse forms 


which means they will also be facing the challenges of program evaluation, can help to 


that come with these efforts. : : 
Hane) a aalat=m(y-\e(=)asme) ma al-tX-m at-iniela <aula 


Indeed, teams in charge of improvement networks advancing productive change. We expand 


often must, for example, manage the development raeret 
on an emerging literature that has begun 
of multi-level intervention systems; support a 


community of diverse, often geographically dispersed to explore how evaluation and continuous 


educators; and navigate a turbulent policy and funding improvement methods can be brought 
environment, among other core tasks.” Such conditions 


together productively.4 


require networks to learn quickly by studying their 
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Evaluators bring expertise in 
gathering, interpreting, and 
supporting the use of evidence, 
which is central to vitalizing 
continuous improvement. 


Many of the tools, techniques, and practices developed 
within the field of program evaluation can help 
network leadership teams as they learn to improve.” 

A long tradition of scholarship has explored how 
evaluators can help educational organizations develop 
their interventions and theories of impact, test and 
refine their approaches, and assess the extent to which 
they are achieving their long-term goals. In addition, 
evaluators bring expertise in gathering, interpreting, 
and supporting the use of evidence, which is central 

to vitalizing continuous improvement. However, we 
also recognize that evaluation can be experienced by 
those engaged in innovation and improvement work as 
counterproductive to their efforts.° This divide can be 
bridged when those with evaluative expertise become 
authentic partners with improvement leaders and 
fully align their analytic efforts with the improvement 


activities and social contexts of the network. 


Conceived and enacted in this fashion, evaluation 
becomes a servant to improvement efforts. This 
approach has the potential of enhancing a network’s 
internal learning processes that in turn lead to a 
greater likelihood of positive impact. It also provides 
capacity to generate over time a base of practical 
knowledge for improvement relevant to education 


more generally. 


This paper starts by introducing improvement 
networks and a conceptual framework highlighting 
key dimensions of their work. Then it describes an 
integrated perspective on informing improvement 
that combines approaches from various evaluation 
methods. We introduce the term “analytic partner” as 
a way to describe the work that individuals or groups 
undertake as they engage with network leaders in 
enacting this integrated approach. We explain the 
challenges that leaders of improvement networks 
face and how analytic partners might structure their 
roles accordingly. This leads, in turn, to a discussion 
of the knowledge, skills, and dispositions that may be 
especially important for analytic partners. We draw 
on our direct experiences working with improvement 
networks, existing writing in the evaluation field,’ and 
the advice and opinions of prominent evaluators and 
scholars who study the use of evidence and joined in 


this inquiry with us. 
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Improvement networks are becoming increasingly 
popular in education and are seen as a high-leverage 
strategy to solve previously intractable problems.* 
Educational innovators recognize the need to organize 
these networks as broad coalitions of diverse actors.° 
These coalitions can take various forms, such as 
Networked Improvement Communities, Collective 
Impact Networks, or Aligned Action Networks.'° Many 
research practice partnerships (RPPs) have emerged in 


this area as well." 


IMPROVEMENT NETWORKS IN 
ACTION 


Two characteristics distinguish improvement 
networks. First, participants commit to advancing a 
shared goal. Unlike other educational networks—such 
as sharing networks or communities of practice— 
improvement networks are communities of common 
accomplishment that come together to make 
measurable progress on a valued outcome.’ Second, 


as they seek to progress, improvement networks 


take a “learn as you go” orientation. That is, instead 

of beginning their work with a rigid commitment 

to a pre-defined program design or set of practices, 
improvement networks expect to study their problem, 
experiment with potential solutions, and continuously 


refine their interventions.!*! 


Improvement networks aim to develop over time a robust 
body of evidence necessary to achieve better outcomes 
across varied contexts and student populations."* As 
network’s mature and their span of influence expands, 
interventions may become more adaptive in seeking to 
respond to diverse local conditions.'” Researchers have 


referred to this network process as developing capacity for 


adaptive integration.'*" 


i This same spirit is shared by efforts advanced under the umbrella of Design 
Based Implementation Research. Source: Penuel, W.R., Fishman, B.J., Cheng, B., 
& Sabelli, N. (2011). Organizing research and development at the intersection of 
learning, implementation, and design. Educational Researcher, 40(7), 331-337. . 


ii A similar argument exists in implementation science in which many social 
interventions are intrinsically interactive in character and must integrate 
productively with local contexts to achieve their goals. Source: Fixsen, 

D.L., Blase, K.A., & Fixsen, A.A.M. (2017). Scaling Effective Innovations. 
Criminology and Public Policy, 16(2), 487-499. https://doi.org/10.1111/1745- 
9133.12288. 
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THE THREE-LEVEL NESTED MODEL 


As a consequence of this adaptive, learning orientation, 
improvement networks have distinct knowledge 
needs that evolve over time. Early on, a network's 
ideas about how to improve are often tentative. 
During this phase, networks may draw on existing 
scholarship, the insights of their members, and locally 
conducted inquiries to develop a working theory of 
improvement. However, in the beginning, this theory 
is provisional. Over time, networks develop their own 
internal analytics and data infrastructure to refine 
this working theory and test their conjectures about 


improvement across a variety of settings. 


Accompanying these efforts, networks will 
simultaneously engage in two parallel streams of 
activity. In the first of these, networks attempt to foster 
a cohesive, collaborative learning community and 
manage the social dynamics of their members toward 


this end. This collection of people, organizations, 


and their work as an improvement network forms 
the improvement enterprise, and supporting the 
enterprise and organizing its efforts is essential for 
accomplishing a network's aims. On the second 
track, networks seek to navigate their environmental 
contexts—the often complicated financial, political, 
and organizational dynamics in which they are 
embedded—which can be highly consequential for a 
network's success, its sustainability over time, and the 


scalabilty of the interventions that evolve within it. 


Figure 1 offers a visualization of the nested relationship 
among these core network activities. The working 
theory of improvement is developed and enacted 

by an improvement enterprise, which, in turn, 

operates in particular environmental contexts. Like 
many frameworks, this three-level nested model 

does not attempt to be comprehensive. Instead 

it highlights a key set of concepts and serves as a 
resource for thinking and reasoning about the work of 


improvement networks. 


Figure 1: The Three-Level Nested Model 
of Improvement Networks 


ENVIRONMENTAL 
fefe) > @ ES 


IMPROVEMENT 
ENTERPRISE 


WORKING THEORY 
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We searched existing literature for ways in which 
evidence could be employed to help networks iterate 
on their theories of improvement, manage their 
enterprises, and navigate their contexts, and found 
considerable work to draw upon. However, only 

by bringing together multiple inquiry traditions— 
developmental,'” formative,'* and summative 


evaluation'’— could these three domains of activity 


be fully addressed. In addition to these core evaluative 


approaches, other types of analytic activity that can 
enhance the work of improvement networks include 
design thinking,” systems thinking,”' improvement 


science,” and direct analytic capacity building. 


These various techniques can be joined together 

in an integrated analytic approach to supporting 
improvement networks that generates useable 
Evidence for Improvement (EFI). We refer to those 
who use these techniques as analytic partners 


to improvement network leaders—a term that is 


shorthand for a broad ensemble of core analytic 
functions, relationships, and organizational capacities 
needed to inform improvement. We use the term 
partner instead of evaluator so as to better signal the 
deeply collaborative and goal-aligned relationship 
of analysts informing practicing improvers. In many 
instances, this relationship may be more complex 
and varied than a single partner, but the term offers 


simplicity in communication. 


We use the term partner 
instead of evaluator so as 

to better signal the deeply 
collaborative and goal-aligned 
relationship of analysts 
informing practicing improvers. 
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In what follows, we leverage our three-level nested 
model of improvement networks to elaborate key 
analytic needs of improvement networks and describe 
techniques and approaches that analytic partners may 


use to meet them. 


DEVELOPING AND ITERATING 
ON A WORKING THEORY OF 
IMPROVEMENT 


The working theory of improvement at the center 
of the three-level model represents a network's best 
current ideas of how to accomplish its aims and the 
concrete interventions derived from these ideas. The 
theory is meant to guide and structure a network's 
thinking as it makes practical improvements. 
Importantly, as noted earlier, theories of improvement 
are provisional early on, so they undergo continuous 
adaptation and refinement as evidence is gathered 
from cycles of change. The development, refinement, 
and elaboration of an improvement theory occur 
iteratively and in a non-linear fashion. However, 

for the purposes of simplicity in explanation, we 
discuss these processes as occurring in two phases: 


(1) exploration and (2) testing and adaptation. 


EXPLORATION 


Improvement networks commonly target complex, 
multifaceted social problems that lack a known 
solution.” Under these conditions, it can be 
counterproductive to become too attached to specific 
change ideas early on. Instead, an improvement 
network will often study its problem, engage the 
users that encounter the problem at ground-level, 


conduct a thorough investigation of the larger system 


factors that influence network activities, and consult 
scholarly research on the topic. This inquiry process 
helps networks develop initial ideas about how to 


make progress.” 


Several inquiry approaches are a particularly good fit 
for improvement networks in their exploratory work. 
First, developmental evaluation offers an orientation 
and some useful tools and methods that are ideally 
suited to helping networks as they search for and 
iterate on their early ideas. Developmental evaluation 
is meant to help innovators tackle social problems 

in complex, unpredictable environments.” Its name 
comes from the fact that it is designed to inform 

the development of social innovations” in which 
development refers to deep, substantial changes to an 
intervention, rather than tweaks or minor revisions.” 
Developmental evaluation positions evaluators as 
authentic collaborators, often embedding them 

as members of a project team. The developmental 
evaluator’s task is to serve as a “critical friend” by 
bringing evidence to bear that can help innovators 
rethink their work as they are in the process of doing 
it. Among other features, developmental evaluation 
emphasizes the importance of (1) close working 


relationships between evaluators and practitioners, 


Developmental evaluation 
positions evaluators as 
authentic collaborators, often 
embedding them as members 
of a project team. 
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Without taking a systems lens, 
improvement networks might 
overlook key dynamics that will 
inhibit or benefit their efforts. 


(2) shared input into the design of the evaluation, and 
(3) rapid feedback.” 


Building on this developmental evaluation 
orientation, a more specific set of inquiry techniques 
often prove especially valuable for networks as they 
develop their working theory of improvement. 

First, techniques taken from the field of design 
research” —which emphasize understanding the 
experiences of individuals who are closest to the 
problem and engaging a broad set of stakeholders 

in the design process—may be especially valuable 

for networks as they probe their understandings 
about the specific problem they are addressing. 
During this phase, members of an improvement 
network might interview frontline workers, such as 
teachers, principals, or students, to understand their 
challenges. In addition to these “empathy” interviews, 
network members may create maps that precisely 
represent the steps in a key process or engage in other 
forms of qualitative study. Such work helps in the 
design of interventions that are truly appropriate for 
their contexts, and helps to avoid “solutionitis”—the 
common tendency to implement the next silver bullet 
in education without assessing the extent to which 


that idea is truly appropriate and effective.*” 


In addition to techniques from design research, 
analytic partners can support their networks in 

this stage by using tools from a systems thinking 
approach.”’ These tools (e.g., system maps, stock and 
flow diagrams) are designed to help users develop 

a broader lens so that they can see how previously 
unknown factors might influence their work. In the 
context of education, these factors might include state 
and district policies that hinder or facilitate the work 
of a network, or other educational programs that may 
interact with a network's plans.” Without taking a 
systems lens, improvement networks might overlook 


key dynamics that will inhibit or benefit their efforts.” 


Finally, the central goal of the exploration phase 

aligns well with guidance found in the practice of 
theory-based summative evaluation.** This evaluative 
strategy emphasizes the development of a conceptual 
model that represents the causal linkages between 
activities and intended outcomes of an intervention, 
the creation of indicators and measures tied to that 
model, and the explicit and systematic testing of those 
linkages. Similar theory development objectives can be 
seen in improvement networks around the use of tools 


such as conjecture maps* and driver diagrams.” 


TESTING AND ADAPTATION 


Over time, the needs of improvement networks change. 
Improvement practitioners seek to test elements of their 
initial working theories of improvement to refine their 
change ideas. Through this work, which will often begin 
in a single or small number of sites, practitioners will seek 
to learn whether some emergent form of the intervention 
can in fact be taken up, implemented well, and generate 
some evidence about efficacy. In simple terms, the goal is 


to determine Can we get this to work somewhere? 
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An initial demonstration 

of efficacy is an important 
first step toward the larger 
overarching aim of developing 
a set of interventions with 
demonstrated effectiveness 
across diverse contexts. 


At this point, evidence on this question may be largely 
qualitative in form. Regardless, an initial demonstration 
of efficacy is an important first step toward the larger 
overarching aim of developing a set of interventions with 


demonstrated effectiveness across diverse contexts.*” 


Once initial evidence of progress has emerged in 

one or a small number of places, networks seek to 
spread these interventions more widely. Key tasks for 
improvement networks in this stage are: (1) supporting 
frontline workers as they conduct further testing of 
change ideas and document these processes to create 
useable knowledge for the network as a whole; (2) 
building a system of practical measures attached to 
the core elements of their working theory because, 

as efforts expand in scale, more formalized methods 
become essential for learning from the work; and 

(3) assessing more explicitly the hypothesized links 
between activities and outcomes in order to drive 


further revisions to their improvement efforts.** 


The EFI approach combines several inquiry techniques 
that can help improvement networks as they refine and 
seek to validate their practices. Analytic partners may 


benefit from drawing on methods found in both 


formative and summative evaluation models.*”' 


Formative evaluation focuses on (1) understanding 
how an intervention is enacted in practice, (2) learning 
from the challenges and successes of that enactment, 
and (3) making appropriate changes to the design 

of that intervention or the way its implementation 

is supported.*” What is key here is that the analytic 
partner “treats variation in implementation not 

as a mediating variable but as a crucial source of 
information." This formative perspective also 
highlights the fact that variation in performance is 
the natural state of affairs. Understanding the sources 
of this variation is key to advancing the effective 
adaptations that may be necessary to assure quality 


outcomes occur reliably as the initiative spreads.” 


While formative evaluation is intended to help 
designers make significant adjustments to their 
interventions, summative evaluation is designed to 
assess the impact of those interventions. It directs 
networks to examine the hypothesized connections 
between the practices they advocate for and the 
outcomes they target. In general, summative 
evaluation (sometimes called impact evaluation) 
assesses how well an intervention accomplishes its 
stated goals; it is an assessment of the efficacy of the 
intervention and its impact on the people who took 


part in it. 


It is important to recognize that the interventions 
being developed by improvement networks are 
typically neither simple nor discrete. Researchers have 


noted that much of the writings on evaluation assume 


i Evaluation is a broad field that includes diverse approaches not discussed 
in this document. For the purposes of argument, we characterize formative, 
summative, and developmental evaluation as distinct approaches while fully 
recognizing that in practice they may overlap with each other and with other 
evaluation models. 
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that the intervention has an “atom-based” character.“ 
Basically, the intervention is discrete and well specified 
and can be introduced to most any context with little 
demands on participants for individual learning 

and organizational change: It can literally just be 
“added in” In contrast, most educational and social 
interventions have a more interactive relationship with 
local context. Network members make adaptations to 
their interventions based on particularities of context, 
and these adaptations may become more numerous as 
a network scales. On balance, some adaptations will 

be more effective than others in particular settings. 

So, learning from this variation in performance is an 


essential task of an analytic partner. 


This interactive and variable character of interventions 
demands a transformation of the traditional 
summative evaluation question Does it work? While 
the summing up of the evidence from an improvement 
network remains an important issue, the summative 
question now becomes What works, for whom, and 
under what conditions? To the point, a growing body 
of evidence indicates that most interventions have 
highly variable effects in which they work well in some 
places and not at all in others.* Researchers report that 
the standard deviation in the size of an intervention’s 
effects may often be as large, and sometimes even 
larger, than the average effect. Moreover, there is 
evidence that these study findings may underestimate 
the propensity for null effects that would occur should 


the interventions scale more broadly.” 


Consequently, in the testing phase, the efforts of the 
network must be viewed as a set of interventions, 
being adapted in varied contexts, with differing levels 
of effectiveness likely to occur. Learning about the 


contours of this variation and the factors driving it, 


The analytic partner needs to 
figure out, within the context of 
a given improvement network, 
some practical ongoing way to 
generate credible and useable 
evidence, context by context 
and for different sub-groups of 
students, so that the network’s 
progress toward its measurable 
aims can be captured. 


as opposed to creating a single number indicator of 
network effectiveness, is the summing up task for 

the analytic partner. Network leaders need regular 
feedback about What is working and what is not, and 
where and for whom? This knowledge is the fuel for the 


next round of continuous improvement. 


Therefore, the analytic partner needs to figure out, 
within the context of a given improvement network, 
some practical ongoing way to generate credible and 
useable evidence, context by context and for different 
sub-groups of students, so that the network's progress 
toward its measurable aims can be captured. The 
design for generating such summative evidence may 
take different forms in different networks, which will 
require of the analytic partner creative blending of 
pragmatic and opportunistic thinking. The data design 
may be based on comparing baseline results in each 
site (ie., historical outcomes) to subsequent changes 
over time—the core mechanism used, for example, in 


most process improvement studies. 
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The mechanisms put in place 
for the summing up of evidence 
must be dynamic like the core 
work of improvement itself. 


Alternatively, it might involve local matching of students 
and sites to form a comparison group,” or it may draw 
on statistical methods such as value-added strategies or 
propensity score analytics to create the comparison. In 
some cases, local randomized experiments may actually 
be possible. Lottery studies on the effects of individual 


charter schools are an example. 


Whatever choice is made, in the context of continuous 
improvement, summative evaluation is not meant to 
be a design used for a one-time study. The efficacy of 
a network's efforts and the contours in its variation 
will change over time. Consequently, the mechanisms 
put in place for the summing up of evidence must be 


dynamic like the core work of improvement itself. 


In sum, though tools drawn from formative and 
summative evaluation play a central role for networks 
in the “testing and adaptation” phase of their work, 

the conceptual frames from both developmental 

and theory-based evaluation remain central as well. 
Aligning with the underlying ethos of continuous 
improvement, the EFI framework is about learning 

as you go. It is an inquiry perspective that informs 
practitioners as they try things out and constantly pays 


attention to results, including variation in and driving 


factors of outcomes, and in the accrual of evidence to 


guide further cycles of testing and revision.” 


BUILDING CAPACITY FOR INQUIRY 


In addition, embedding evaluative thinking and 
methods into the direct work of network participants 
can enhance their capacity to learn rapidly from 

their own efforts.*° Thus, another important role for 
an analytic partner in vitalizing a network's learning 
about its own efforts is through building the capacity 
of network members to engage in systematic inquiry 
about their own work. In this regard, analytic partners 
may train and coach network members directly 
engaged in improvement work. In these efforts, 
analytic partners could focus on three broad forms of 


direct capacity building. 


First, analytic partners could help network members 
learn about and use the tools of improvement 
science that are designed to help practitioners test 
and document their practices in order to refine them 
and improve.*! Improvement science, for example, 
emphasizes the use of plan-do-study-act (PDSA) 
cycles, an adapted version of the scientific method 
designed for frontline workers. PDSAs can help 


practitioners systematically study their own practices 


Aligning with the underlying 
ethos of continuous 
improvement, the EFI 
framework is about learning as 


you go. 
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and make targeted refinements to these practices based 


on their inquiry. 


Second, improvement science also urges practitioners 
to develop a system of measures to track how their 
interventions actually operate in the field and to 
provide the necessary data resources to continually 
assess the causal linkages assumed in the network's 
working theory of improvement.” The guiding 

spirit of improvement is “definitely incomplete and 
possibly wrong.” Regularly examining evidence from 
a system of measures is critical to enlivening this 
mantra. By supporting network members in the use 
of improvement science tools, analytic partners can 
enhance the rigor with which the network studies 

its own practices, builds a body of evidence around 
their use, and makes informed decisions about how to 


adjust its work. 


While such a system of measures, once produced, 


may appear relatively straightforward, the design, 


By supporting network 
members in the use of 
improvement science tools, 
analytic partners can enhance 
the rigor with which the 
network studies its own 
practices, builds a body of 
evidence around their use, 
and makes informed decisions 
about how to adjust its work. 


testing, and refinement of the measurement system 

is a complex technical task. It requires a blending 

of content knowledge about the problem-solution 
space; a knowledge of local contexts coupled to 

an engineering orientation to make the measures 
practical, light, and lean; and statistical acumen to 
examine and refine the predictive validity of the overall 
measurement system. Few, if any, school- or district- 
based educators will have the capacity to guide this 
work on their own. This is a specific instance in which 
the technical skills of an analytic partner come to the 
fore amid the broad contours of expertise needed to 


vitalize improvement networks.*° 


Third, analytic partners can also directly enhance the 
capacity of networks to engage in systematic inquiry 
by helping them build analytic infrastructure.”* Based 
on experience coaching and supporting educational 
improvement networks, we have learned that these 
networks often face challenges developing the routines, 
roles, and tools required to efficiently gather, manage, 
analyze, and interpret data. Many educators receive 
only minimal training in collecting data systematically 
and analyzing results. As a consequence, improvement 
network leaders often need guidance on how to 
undertake these activities, which are essential to testing 
changes in continuous improvement. Analytic partners 
can help networks develop greater capacities related to 
data infrastructure in a variety of ways. For example, they 
might help networks develop efficient social routines 
for gathering and analyzing data, advise networks on 
how to recruit and train appropriate analytic personnel, 
or recommend technological systems for the storage, 


analysis, and presentation of data. 
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ESTABLISHING AND MANAGING AN 
IMPROVEMENT ENTERPRISE 


The second level of our three-level model focuses on 
the improvement enterprise—the group of people and 
organizations that make up an improvement network. 
Improvement enterprises typically include a diverse 

set of members that help the network leverage varied 
knowledge pools, accelerate its collective learning, and 
broaden its impact.** Improvement networks often 

cross organizational boundaries (e.g., school districts, 
schools) and bring together practitioners who do not 
typically have the opportunity to regularly interact with 
one another. Research tells us that such networks have 
tremendous potential to accelerate learning and foster 
innovation.** However, at the same time, their relatively 
loose authority structure and dispersed nature can make 
coordination difficult. In order to promote coherent 
collective action, networks typically have some form 

of leadership or hub structure, along with a central 
group convening, communicating with, supporting, and 


orchestrating the learning of other members. 


Leaders charged with managing and nurturing the 
improvement enterprise face a complex challenge. 
They must (1) identify and recruit new members, 

(2) help to establish and maintain collaborative norms, 
(3) design and execute meetings and convenings in 
which the network comes together, and (4) promote 
the development of social connections and trust 
among network members. At the same time, they are 
asked to facilitate the collective learning of the network 
by (1) sharing the insights that emerge from the work 
of individual members and smaller teams, (2) drawing 
in external knowledge from research and scholarship 
as needed, and (3) modeling and promoting a learning 


culture in which risk taking and experimentation feels 


Improvement networks often 
cross organizational boundaries 
and bring together practitioners 
who do not typically have the 
opportunity to regularly interact 
with one another. 


safe. All the while, network leaders need to keep their 
members focused on the ultimate goal of improving 
outcomes in their individual workplaces and across the 


entire network.’ 


Given the complicated social structure of improvement 
networks and the challenge of leading this type 

of organization, analytic partners can provide 
considerable value by generating evidence on network 
dynamics and reporting what they observe to a 
network's leadership team. As network leaders balance 
the need to set direction, keep the community engaged 
and energized, and manage members’ learning, 

they may value assistance sensing problematic 


developments within their enterprise and responding 


Analytic partners can 

provide considerable value 

by generating evidence 

on network dynamics and 
reporting what they observe to 
a network’s leadership team. 
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accordingly. Analytic partners with an EFI orientation 
may, for example, explicitly map the emerging patterns 
of social interaction developing in the network 
through social network analysis (SNA).°* This can 

help leaders to identify problematic social disconnects 
among their members, such as a lack of social ties 
between sites within the network, and provide targeted 
supports to create those connections, among other 
uses. In addition to SNA, analytic partners might also 
conduct climate surveys or observe network meetings 
as a means of assessing the learning and engagement of 
network members. This type of organizational analysis 
serves as a source of systematic feedback that can 
assist managers to better understand the operational 


dynamics of their enterprise. 


For example, the Carnegie Foundation for the 
Advancement of Teaching, in partnership with 
colleagues at the University of Pittsburgh, has 
developed an initial network health framework,” a 
matching survey, and a set of fieldwork protocols. 
This framework identifies the key features of an 
improvement network, articulates the relationships 
among these features, and identifies measures that 
can assess them. It focuses on network culture, 
collaborative activity, and members’ perceptions of 
learning opportunities, among other features, by 
generating information tied to the framework, sharing 
it with network leadership teams, and providing 
opportunities for sense making and interpretation. 
These efforts have helped identify challenging aspects 
of network development and informed network 
leaders efforts to target areas for improvement within 
their enterprise. In general, improvement networks 
aspire to develop and sustain a broad base of social 


learning across participants and contexts around the 


goal of solving a shared problem. The work of analytic 
partners with the improvement enterprise seeks to 
inform network leaders’ efforts to initiate and sustain 


these interactions. 


OPERATING IN COMPLEX 
ENVIRONMENTS 


The third level of our conceptual model is the 
environmental context—the larger economic, 
political, and sociocultural settings in which 
improvement efforts are embedded. Networks come 
into specific contact with these forces as they introduce 
and adapt their interventions to the particularities of 
local contexts. These environmental influences also 
operate more broadly, including, for example, how 
networks seek out funding, align their efforts with the 
policies and standard operating procedures of existing 
organizations, and attend to leadership support 
especially in times of senior leadership transition. To 


make matters more complex, networks commonly 


Improvement networks aspire 
to develop and sustain a broad 
base of social learning across 
participants and contexts 
around the goal of solving a 
shared problem. The work 

of analytic partners with the 
improvement enterprise seeks 
to inform network leaders’ 
efforts to initiate and sustain 
these interactions. 
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The work of establishing 

and managing the interface 
between networks and their 
environments requires that 
network leaders enact and 
interpret environmental 
dynamics and make decisions 
about selectively bridging 
and buffering environmental 
influences. 


include members from various different organizations 
(e.g., different school districts), which requires 
networks to interact with not one but many different 
local contexts. Indeed, researchers have argued 

that it is environments (and not activities within 
networks) that most often undermine improvement 
activities within networks and weaken their viability.” 
Moreover, these network dynamics become more 
complex—and more challenging—in contexts that 
have the greatest need and potential to benefit from 
network-based improvement: low performing schools 


in turbulent urban district and policy environments. 


Consequently, the work of establishing and 
managing the interface between networks and their 
environments becomes as important as managing 
activity within networks, requiring that network 
leaders enact and interpret environmental dynamics 
and make decisions about selectively bridging and 


buffering environmental influences.” Important 


aspects of networks’ environmental context include: 
(1) the community (or communities) in which 
network initiatives seek to operate, (2) the research 
field that is relevant to the network’s domain of 
practice, (3) the policy arenas to which the network 
must attend, and (4) shifting funding priorities 


among governments and philanthropies. 


Analytic partners might help network leaders negotiate 
these environmental factors in several ways. First, 
they might help leaders build greater awareness 

that environmental factors matter. Network leaders 
may not be able to fully attend to how environmental 
factors influence their work as the demands of 
operating their organization and/or supporting 

the successful enactment of their change ideas 

are naturally privileged. Second, analytic partners 
might help network leader's map and monitor 
important elements of their environmental context. 
Systems mapping® or other forms of environmental 
scanning™ may be particularly useful for helping 
networks identify the actors in the environment upon 
which they depend and interrogate the nature of 
those dependencies. Finally, analytic partners might 
help network leaders strategically manage their 
relationships with external actors by buffering their 
influence,® building connections, or trying to actively 


intervene in order to shape their context. 


SCALING WITHIN COMPLEX 
ENVIRONMENTS 

Environmental considerations become especially 
germane as networks seek to scale their efforts 
beyond initial testing and co-development roles. 
Several considerations shape network efforts in this 


regard. First, successfully scaling requires networks 
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Successful implementation 
within any given context is more 
likely if networks ensure the 
organizations within which they 
operate (i.e., states, districts) 
have policies and practices 

that are compatible with the 
network’s change ideas. 


to consider demand-side diffusion strategies, such as 
ensuring their interventions are easily understood, 
perceived to be advantageous, and favored by local 
opinion leaders. Second, successful implementation 
within any given context is more likely if networks 
ensure the organizations within which they operate 
(ie., states, districts) have policies and practices that 
are compatible with the network's change ideas. Finally, 
as networks seek to grow and sustain their efforts, 
they often must cultivate new funding opportunities 
while simultaneously working to maintain their 
current sources. Each new funding source may add 
additional constraints and/or directions for the work 


going forward. 


Spreading a complex change effort while maintaining 
the integrity of its core practices is especially challenging 
for networks. As an intervention becomes increasingly 
popular, it may be taken up rapidly but often only 

in a surface-level fashion. The time and processes 


for deep learning among participants about the 


intervention, which was part of the early stage testing 
and development, may receive short shrift. Coupled 
with the various environmental forces previously 
mentioned, local implementations will likely be 

quite variable. For all of these reasons and more, the 
analytic partner should begin with the assumption 
that, as the intervention moves out over time into 
other contexts, local actors will change it. Learning 
from these sources of variation becomes another key 
objective for the analytic partner. Many adaptations will 
likely be maladaptations, but some might be genuine 
improvements. The network's learning goal over time 
is to continue to reshape the overall distribution of 
outcomes observed—truncating the more negative 
consequences and gradually moving the overall 
center of gravity in a more positive direction. This 
requires ongoing attention to the changes occurring 
to the intervention and to the variation in outcomes 


resulting—here again core work for the analytic partner. 


In sum, the information gained by engaging with a 
partner not directly involved in the work is likely to 

be highly valuable in helping improvement leaders 
monitor, interpret, and adapt to environmental 

factors in ways that both sustain the enterprise and 
support the scalabilty of outcomes. In the same way a 
marketing department supports commercial firms in 
reacting to market trends, an analytic partner may help 
an improvement enterprise respond better to nuances 
of context as well as to changes in their policy, funding, 


and regulatory environments. 


Table 1 summarizes the needs of improvement 
networks and the matching types of analytic support 


that analytic partners could provide. 
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Table 1: How Could Analytic Partners Meet the Needs of Improvement Networks? 


Improvement networks 
SSEK tO... 


So they need help ... 


IMPROVEMENT 
ENTERPRISE 


ENVIRONMENTAL 


CONTEXTS 


20 


Explore new practices, iterate on early 
change ideas, and develop reliably 
effective interventions in their context. 


Avoid solutionitis by deeply studying their 


problem and the system factors that 
produce it. 


Systematically study and iterate on their 


interventions through disciplined inquiry. 


Manage social dynamics across network 
members in different contexts. 


Accelerate social learning across 
the network. 
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Analyzing their efforts through 
a developmental, formative, and 
summative lens. 


Understanding the true needs of their 
users and taking a systems view of 
the problem. 


Building their own capacity to use 
improvement science and developing 
an infrastructure that allows them to 
efficiently analyze data. 


Understanding the nature of 
participation, engagement, and social 
learning occurring across the network. 


Resourcing technical research expertise, 
consolidating learning within the 
network, and creating mechanisms 

for more rapid diffusion of emergent 
knowledge network-wide. 


Sensing salient dynamics within their 
communities, their policy environment, 
relevant fields of academic research, 
and the funding environment. 


Analyzing local adaptations made to 
interventions, understanding why these 
are occurring, and analyzing effects of 
the interventions across 

different settings. 


The preceding section posits that the evolving 
knowledge needs of improvement networks necessitate 
an integrated approach to analytic support that 
includes techniques from various inquiry traditions. 
However, research on knowledge utilization has shown 
that the social conditions surrounding the use of 
evidence are just as important as the evidence itself 
when the goal is to promote substantive interpretation 
and action.” With that in mind, the next item for 
consideration is how analytic partners might define 
their roles and structure their partnership with 
network leaders in order to promote deep engagement 


in evidence for improvement. 


NETWORK LEADERSHIP TEAMS 


Next, we consider how an analytic support aligned 
to the working conditions of network leadership 
teams may be productively designed to enhance the 
social learning of the network. Network leaders may 
have difficulty keeping track of the many factors that 


influence their efforts (such as those introduced in 


the three-level nested model). They will also likely 
experience confirmation bias—a tendency to interpret 
phenomenon in a way that confirms their pre-existing 
beliefs.”” Given these realities, analytic partners can 
provide an essential balancing function. They can offer 
a disciplined yet supportive perspective that helps 
network leaders interrogate their own ideas about the 
activities occurring in their network. In particular, 
three key working conditions of network leadership 


teams may make it especially challenging for leaders to 


Analytic partners can offer 

a disciplined yet supportive 
perspective that helps network 
leaders interrogate their own 
ideas about the activities 
occurring in their network. 
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engage in deliberate and intentional decision-making: 
(1) the need to constantly and rapidly learn from 
practice; (2) the intense day-to-day demands of their 
work; and (3) the inherent risk, uncertainty, and 
vulnerability of working in an improvement context. 
These conditions, in turn, highlight the value of an 
analytic partner whose primary role is informing the 


work rather than doing (or managing) it. 


THE NEED FOR CONSTANT, 
RAPID LEARNING 


The contexts in which many educational improvement 
networks seek to intervene, such as schools and 
districts, are beset by notoriously complex and 
multifaceted problems.” In these conditions, networks 
strive to learn from their initial attempts at change, 
adapt their work as they go, and continuously improve. 
Researchers have called this feature of network 
leadership the “learning imperative,” which has at least 
two key qualities: (1) it is constant and ongoing, and 
(2) it requires the ability to rapidly access and respond 


to incoming data and evidence.” 


With these two demands on network leaders’ learning 
needs in mind, an analytic partner should be readily 
available for consultation and discussion. This would 
require the embedding of an analytic partner as a full 
member of the leadership team so as to maximize their 
availability, knowledge of the context, and investment 
in the improvement effort. What is most important is 
that they are in constant communication with network 
leaders and thus positioned to help the network 
identify relevant information and draw attention to 


new evidence as it becomes available. 


We recognize that, for many traditionally trained 


researchers and evaluators, acting as a full member 


of an initiative’s leadership team may appear to be 
challenging. After all, many evaluators have been 
trained to be independent, to act at some arms-length 
distance from the intervention, and to see their efforts 
as accountable to external authorities (e.g., funders) 
who seek an impartial report about the intervention’s 
implementation and effectiveness. However, to best 
help networks improve, an analytic partner’s first 
priority should be to support the learning of network 
members. Consequently, for evaluators to maximize 
their contribution to this goal means they must 
challenge the more traditional stance of an evaluator 
as an external and disinterested actor. In fact, the core 
value of an analytic partner is their proximity to the 


improvement activity. 


Taking on this role, however, does not mean 
abandoning disciplined inquiry or failing to bring 
rigorous empirical evidence to bear. In fact, it is 
precisely this set of skills and orientation that makes 
the analytic partner of great value to the improvement 
network. That is, as network leaders aspire to advance 
on the aims they have espoused, analytic partners 
bring forward evidence that network leaders need to 
know and attend to so that they can continue to move 
their enterprise forward. In addition, if the work of the 
analytic partner is well carried out, a comprehensive 
evidence base will develop that is capable of answering 


most any question raised by external audiences. 


The core value of an analytic 
partner is their proximity to the 
improvement activity. 
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If more assurance is needed, an external “analytic 
audit” of these records (see Appendix) can be 
commissioned. Such an audit might work in ways 
analogous to how a corporation's financial records are 


subject to independent external review. 


BALANCING DAY-TO-DAY 
RESPONSIBILITIES 


Network leadership teams, like the leaders of 

other complex organizations, are faced with a 
daunting set of responsibilities. Over and above 
their responsibilities for program development and 
implementation, leadership teams are commonly 
responsible for funding, human resources, operations, 
recruitment, and a variety of other tasks.”* Given 

the breadth of their responsibilities, improvement 
leaders can become fully occupied with the day-to- 
day requirements of running their enterprise. When 
leaders step back from the daily demands in order 
to reflect, learn from their ongoing work, and adjust 
the network's overall direction, they are able to make 


more thoughtful decisions. 


When leaders step back from 
the daily demands in order 
to reflect, learn from their 
ongoing work, and adjust the 
network’s overall direction, 
they are able to make more 
thoughtful decisions. 


The book Leadership on the Line uses the metaphor of 
a dance floor and a balcony to illustrate the importance 
of leaders taking an expansive perspective.” Even 
though a leader may spend some time on the ground 
dancing and participating in an event, they should 
make sure to “get on the balcony” that overlooks the 
dance floor so they can see a larger picture of what is 
going on. Because it can be difficult for leaders to zoom 
out and view their organizations with an expansive 
lens, especially when they are in the midst of a change 
initiative, one of the primary functions of analytic 
partners is to help leaders get on the balcony. Analysts 
can provide value by considering the improvement 
network holistically, keeping its broader goals front 
and center while considering the larger system factors 


that are influencing its work. 


Just as getting caught up in the day-to-day can prevent 
a network's leadership team from reflecting on the 
larger forces affecting its organization, it can also 
cause them to lose sight of their end goals. Therefore, 
another important function of an analytic partner 

is to help network practitioners stay accountable to 
their own goals. This can be done in at least two ways. 
First, analytic partners can serve as the “empirical 
conscience” of a network by focusing members 

on the hypothesized causal connections in their 
working theory of improvement. Are positive changes 
taking place? If so, are the expected causal linkages 
happening? The network has introduced practices that 
they hypothesize will improve certain processes and in 
turn enhance some specific outcomes. Is this sequence 


occurring as expected? 


Second, analysts can reach beyond these empirical 


questions and prompt broader reflective discussions 
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around questions such as To what extent are we 
accomplishing what we set out to do? or To what extent 
are our actions aligned with our purposes? These types 
of questions aim to promote internal accountability”® 
and help network practitioners stay true to their core 


objectives and values. 


RISKY, VULNERABLE, UNCERTAIN 
WORK 


Members of improvement networks are often trying 
to make progress on multifaceted social problems 
with a solution path that is far from clear. In such 
situations, humility, openness, and adaptability are 
valuable assets. Practitioners in improvement networks 
are constantly trying out new approaches that may or 
may not work in the hopes that they will learn quickly 
from failure. Unfortunately, working with traditionally 
trained evaluators may not be a good fit for this 
learning-by-doing mindset. Indeed, commonly “fear 
of failure pervades evaluation” because evaluations 


emphasize the performance of a program above all.” 


As a consequence, the evidence being generated by 
analytic partners and the way that they share that 
evidence plays a central role in helping improvement 
practitioners learn from their work. In these inherently 
vulnerable situations, it is important that practitioners 
and analysts develop trusting relationships. Trust 
eases communication. It creates a sense of safety that 
encourages network members to react to failures 

with curiosity instead of defensiveness.”* Analytic 
partners can promote trust by engaging in long-term, 
collaborative relationships; acting with integrity 

(e.g., following through on commitments); and 
demonstrating personal regard for their clients,” 


among other means. 


Trust eases communication. 
It creates a sense of safety 
that encourages network 
members to react to failures 
with curiosity instead of 
defensiveness. 


Table 2 summarizes how an analytic partner can 
engage the working conditions of a network leadership 
team. Taken together with Table 1, it serves as a 
summary of the integrated EFI approach. These tables 
connect the needs of networks and network leaders 

to specific analytic activities and ways of working 
together that may meet these needs. Our intention 
here is to provide a set of ideas, not a guidebook 

or checklist. Any particular analytic partner will 
selectively draw on these ideas based on their own 
expertise, their assessment of network's needs, and the 
developmental stage the network is in. For the sake 

of brevity, we do not elaborate on the specific analytic 
tools that partners might use, though more detail on a 


select list of tools is available in the appendix. 


In sum, as analytic partners engage in EFI activities, 
their orientation toward their work is goal-aligned 
with the network they are supporting. They draw 

on a diverse set of tools and methods to inform the 
network’s efforts to achieve its aims, with a stake in its 
success and shared accountability for its results. They 
best advance the network's outcomes by conducting 
the most responsive and thoughtful disciplined 
inquiry that they can. This type of goal-aligned role is 


distinct from the traditional evaluator, whose primary 
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responsibility is often to the funder and not the 


grantee. In this way, taking an EFI approach dissolves 
Taking an EFI approach 


the tension that often exists between an evaluator and 


the program staff they are evaluating. Evaluation now dissolves the tension that often 
is not about the program and its staff but rather for exists between an evaluator 

the program and its staff to assist them in advancing and the program staff they are 
their goals. Therefore, the analytic partner and the evaluating. 


improvement practitioner should be unambiguously 


and unabashedly on the same team. 


Table 2: How Could Analytic Partners Support Network Leadership Teams? 


Improvement leadership teams ... So they need ... 


Face a learning imperative Analytic partners that are embedded in the leadership team, 
Help their organization learn quickly from its efforts in orderto regularly available, and capable of providing rapid feedback 
rapidly improve 


Time to reflect and “get on the balcony”* 
Help to stay focused on their end goal and stay internally 
accountable to it 


Analytic partners that emphasize an ongoing, trusting 
professional relationship 


* Heifetz, R., & Linsky, M. (2002). A Survival Guide for Leaders. Harvard Business Review, 80(6), 65-74. 
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This white paper has outlined a framework for how 
analytic support can assist in advancing network 
efforts toward improvements at scale. This section 
will address some of the dispositions, skills, and 
knowledge demands on analytic partners to execute 
well in this role. This analytic work requires a wide 
variety of competencies, so the description that 
follows is not that of a single person, but rather 

of the technical and social competencies required 
of analytic partners more generally. Importantly, 
efforts to apply the EFI framework are still in their 
infancy, and much still needs to be learned about 
how to execute it well in different settings. Though 
the characteristics described in this section begin to 
clarify what is required for this work, they should be 


viewed as provisional and incomplete. 


DISPOSITIONS 
As previously argued, analysts should fully engage 


with improvement practitioners and serve as 
authentic partners in the work of achieving the 
improvement aim.® Being effective in this role likely 
requires an embrace of humility and a collaborative 
mindset. While the analytic partner may bring 
distinct methods and skills to the network hub, 

they are not the knowledge authority. By their 

very nature, improvement networks bring together 
members with distinctive and varied experiences and 
expertise. Because analytic partners join this diverse 
colleagueship as full members, they are true partners. 
In their roles, they need clarity in what they know and 
don't know, and what they can and cannot bring to the 
enterprise, while also remaining open and vulnerable 
to learning through the work. Like network leaders, 
they should be comfortable with ambiguity. As part 
of the network leadership team, analytic partners will 
need to learn their way into how best to support the 


network’s improvement efforts. 
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Knowing when to call attention 
to problematic program 
developments and how 

to frame these comments 
requires the analyst to exhibit 
judgement, tact, and social 
sensitivity. 


SKILLS 


An EFI orientation makes demands on both technical 
and social-interpersonal skills. As a critical friend, 
one of the essential tasks of an analytic partner is 

to raise questions and bring evidence to bear in 

their discussion. Knowing when to call attention 

to problematic program developments and how to 
frame these comments requires the analyst to exhibit 
judgement, tact, and social sensitivity.*! The ability to 
build relationships and sustain rapport with others 
over an extended period of time is also essential 

in partnering with an organization. Additionally, 
facilitation skills will likely be highly important 
because a key task of using evidence for improvement 


is to help groups make sense of data and evidence. 


At the technical level, an EFI orientation requires 
methodological versatility and flexibility. The process 
of helping a network refine and test its working theory 
of improvement emphasizes a strong background 

in traditional social-science research methodology. 
Knowledge of measurement development may 


be particularly important, with an emphasis on 


developing practical measures that are minimally 
intrusive while still predictive of targeted outcomes.” 
To help a network test their theory, analysts will 
likely be well served by a grounding in inferential 
statistics and correlational and quasi-experimental 
methods. These tools allow network members to 
identify associations between the activities they 
pursue, the mechanisms for improvement assumed, 
and the outcomes to which they aspire. Beyond these 
quantitative skills, an analytic partner will also likely 
require qualitative and mixed-method research skills 
such as interviewing, ethnographic observation, and 
survey development. These methods are valuable 
during the early stage of network formation in 
deepening understandings about the problems to 

be solved and the system forces shaping them. They 
are also likely to be called upon to help a network 
understand its expanding enterprise and its broader 


ecological context. 


KNOWLEDGE 


Analytic partners are also best positioned to provide 
value when they bring relevant content expertise 

to their work. Such expertise lends credibility and 
helps analytic partners to ask better questions 

about the work of the improvement enterprise and 

to see important connections. In the context of 
improvement networks, multiple forms of content 
expertise may be important. First, knowledge of 
improvement methodologies, such as design thinking 
and improvement science, is essential. Knowledge of 
improvement science, for example, entails familiarity 
with the tools outlined in classic texts on improvement 
thinking™ and an understanding of the experimental, 
learning mindset practitioners aim to adopt in order to 


use improvement tools at their full potential. 
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Second, knowledge of a network's specific problem of 
practice and context will likely also be beneficial. Such 
knowledge demands will differ substantially based on 
the particular focus of the network (e.g., early grades 
literacy, middle-school mathematics, etc.). Beyond 
content expertise, training in systems thinking and its 
associated tools (systems maps, process maps), skill in 
the analytics of variation in performance, and a general 
knowledge of educational organizations (e.g., schools, 
districts) may be very helpful for an analyst trying to 


understand how a network interacts with its contexts.* 
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In this paper, we have introduced the role of an 
analytic partner in an improvement network and 
described how an evidence for improvement 
framework can productively guide their work. At 

the center of the framework are ongoing systematic 
inquiries that inform the network as it iterates through 
its working theory of improvement. Drawing on 
writings about how a scientific professional learning 
community develops among network members,” 
the EFI framework also highlights the importance of 
analyzing the social connections and collaborative 
norms emerging in the improvement enterprise. 
Finally, the EFI framework pushes analytic partners 
to reckon with the environmental contexts in which 


improvement efforts are embedded. 


The approach presented here builds on a rich practice 
of evaluation techniques developed over the last 

50 years. It draws selectively from these traditions 

to craft a coherent and integrated framework that 


can guide future evaluative efforts for continuous 


improvement networks. We see the primary 
contribution of this paper as making more visible 

the connections between (1) longstanding practices 
in program evaluation and (2) the gathering and use 
of evidence for continuous improvement carried out 
through structured networks. In so doing, we hope to 
spark interest and scholarship around such efforts in 
the educational field. 


We encourage future research and writing to examine 
and investigate the usefulness of the practices we 
propose here so that they can be refined and better 
understanding of how working analysts might use them 
can be developed. It is only through such inquiry that 
we as a field can advance our understanding of how to 
best support networks as they engage in the challenging 


work of continuous learning and improvement. 
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APPENDIX: AN EMERGING TOOLKIT FOR EFI 


This section identifies a preliminary set of tools that 
analytic partners might use to support improvement 
networks. within the context of of the three-level 
analytic model. Figure A-1 lists the tools that are 
briefly described. 


Figure A-1: Tools That Can Support 
Improvement Networks 


1.. Enhancing hub capacity 
for inquiry 
a. Empathy interviews 
b. Systems maps 
c. PDSAs 

2. Theory and measurement 
development 

3. Analytic infrastructure 
capacity building 

4. Analytic audits 


WORKING THEORY 
OF IMPROVEMENT 


(a) 


IMPROVEMENT 
ENTERPRISE 5. 


(LEVEL 2) 


Network health assessments 


ENVIRONMENTAL 
CONTEXTS 6. 


(aay) 


Environmental scans 


OVERALL 


(aA ES ee ae)) 


THE WORKING THEORY OF 
IMPROVEMENT 


ENHANCING HUB CAPACITY FOR INQUIRY 


Network leadership teams commonly engage in 
inquiry to understand the systems in which they work 
and the interventions they develop. By nature of their 
training, analytic partners may be well positioned to 
help build a leadership team’s capacity for productive 
inquiry. Three tools that network leadership teams and 
analytic partners may draw on are empathy interviews, 


systems maps, and PDSA cycles. 


Empathy interviews offer a strong tool for gaining 
knowledge from end-users. This technique is adapted 
from the field of user-centered design,” and it focuses 
on understanding people's thoughts, emotions, and 
needs in an effort to determine which interventions are 
best suited to them and their contexts. Systems maps, 
in turn, describe the subsystems most likely to manifest 
themselves in the improvement work itself.* While 
members of a network may know aspects of the system 
in which they work or learn, often individuals struggle 
to see a system in its (approximate) entirety. The 

PDSA cycle provides improvement practitioners with 
a systematic and rigorous way of studying the changes 
being implemented. PDSAs vary in scale and scope, but 
all require practitioners to propose hypotheses (plan); 
attempt and document a change (do); assess the results 
(study); and make decisions about how and whether to 


revise, adopt, or abandon that change (act).*° 
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Analytic partners often have skills and knowledge 
useful in supporting hub teams in each of these three 
tools. Analytic partners with training in qualitative 
research methods can assist members of a network in 
formulating questions for users and analyzing the data 
collected. Similarly, an analytic partner may be well- 
suited to help network members develop a systems 
map so that the network is better able to identify the 
components in their system. Finally, improvement 
practitioners may need help in applying the precise, 
scientific thinking that a PDSA requires to their 


complex daily work. 


THEORY AND MEASUREMENT 
DEVELOPMENT 

At their core, continuous improvement efforts are 
driven by theory-based learning. Participants articulate 
a theory of improvement that connects specific ideas 

in an extended causal chain that ultimately targets 
valued outcomes. They then create a set of measures 
that allows them to make inferences about these 
hypothesized causal linkages. When done well, a 
disciplined approach to theory-based inquiry can give 
practitioners confidence that their specific practices are 
moving them towards their aim. By the same token, 
such an approach can challenge practitioners, convince 
them that their practices are not working, and thus 
spur important revisions. However, constructing a 

set of measures tied to a theory of improvement and 
testing the linkages between these measures is no 


simple task. 


In our experience, practitioners engaging in 
improvement often have trouble with this component 
of the work—especially when they are beginning their 


efforts. Therefore, one potentially valuable activity 


for an analytic partner is to support measurement 
development. The purpose of such support is to 
investigate and help refine (1) the measures that an 
improvement effort is using to assess its theory and 

(2) the techniques the effort is using to make 
inferences based on those measures. Of primary 
importance, an analytic partner might consider how to 
refine a network’s measures so that they are practical 


for use in improvement efforts.” 


ANALYTIC INFRASTRUCTURE 
CAPACITY-BUILDING 

Improvement networks seek to develop their own 
internal capacity to develop measures attached to their 
theories of improvement, assess the links between 
those measures, and revise their theories accordingly. 
In order to accomplish these tasks in an efficient 
manner, networks need an analytic infrastructure— 
that is, the roles, routines, and tools that allow a 
network to collect, manage, analyze, and interpret 
data efficiently. Commonly, improvement networks 
lack these resources, especially early on. Thus, analytic 
partners may choose to spend time providing training 
and tools to network members with the goal of helping 
sustain a high-functioning infrastructure. For example, 
an analyst may develop a data dashboard that displays 
all of a network’s improvement measures in one 
location, provide a network with the software that was 
used to create that dashboard, and instruct network 
members in the use of that software. An analytic 
partner might also help the network study the social 
processes it uses to collect and manage data, identify 
bottlenecks and or redundancies, and assist in the 


design of a new process. 
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ANALYTIC AUDIT 


Beyond helping a network develop its own measures 
or working to build an analytics infrastructure, 

an analytic partner may provide a review and/or 
certification function for a network's testing, theory 
development, and outcome assessment processes. In 
an analytic audit, an analytic partner would assess 

the practicality and predictive power of a network’s 
measurement system. He or she would also assess the 
nature by which a network had tested links between 

its activities and outcomes, and verify that any 

claims the network had made about the effects of its 
activities were supported by its data. Depending on the 
network's needs, an analytic partner might conduct the 
audit, or he/she might prepare the network for an audit 
conducted by an external group such as an evaluation 
firm or government body. Though it might provide 
valuable formative feedback to a network, the primary 
purpose of such an audit as outward facing. If an 
analytic partner and/or an external firm undertaking 
an audit could verify that the measurement system 
and testing strategy of a network was of high quality, 

it would provide that network with credibility in its 


broader environment. 


THE IMPROVEMENT ENTERPRISE 
NETWORK HEALTH ASSESSMENT 


Networks are a distinct organizational form with 
unique advantages and disadvantages. Given the 
potential and the challenge of working through a 
network form, analytic partners can provide value by 
assessing network's health and sharing this assessment 


with network members. As described in the main body 


of the paper in the section called “Establishing and 
managing an improvement enterprise, Carnegie has 
been developing a suite of network health assessments 
that identify key aspects of network functioning 

and help network leadership teams tailor their 


organizational design and communication strategies. 


THE ENVIRONMENTAL CONTEXT 
ENVIRONMENTAL SCAN 


Environmental scanning is a common practice in 
business and healthcare in which organizations assess 
features of their environment (e.g., competitor's 
behavior, sector-level and global trends, policy 
changes), derive implications for their work, and 
respond strategically. Environmental scanning 

allows organizations to identify factors that may act 
as barriers or facilitators of their work, help them 
anticipate changes occurring in the broader sector, and 
guide them towards actions that will be most effective 
in their contexts.”’ By using environmental scanning, 
analytic partners can help improvement networks 
adjust to the constraints in their environment, take 
advantage of facilitating opportunities, and/or work 
to change aspects of their environment that are 


challenging (e.g., by lobbying or political organizing). 


OVERALL 
IMPROVEMENT REVIEWS 


An improvement review is a process by which an 
external group of experts performs a comprehensive 
assessment of an improvement network at a given 


point in time. The purpose of this review is to provide 
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an improvement effort with formative feedback and theory of improvement and associated measures, the 
advice. Members of an improvement review panel may development and health an improvement enterprise, 
include improvement science experts, content area and how the effort is interacting with its local and 


experts, and/or peers that work in other improvement __ regional contexts. An analytic partner can help arrange 


networks. Importantly, an improvement review may improvement reviews, participate as a member of 
cover all three levels of our nested model. During a an improvement review panel, and help a network 
review, panelists may discuss the network’s working leadership team make sense of the feedback that comes 


out of a review process. 
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